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ABSTRACT 


The  techniques  developed  in  recent  years  to  deal  with 
stationary  processes  are  presented  in  a manner  suited  to  the 
purposes  of  those  who  use  Fourier  analysis  or  its  variations 
for  the  analysis  of  their  data.  Necessary  probability  background 
is  provided  and  the  spectral  analysis  of  discrete-time  and 
continuous -time  processes  is  carried  out  in  detail  with  a number 
of  examples  to  illustrate  the  applications  of  the  theory. 
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STATISTICAL  ANALYSIS  OF  A STATIONARY  PROCESS 


by 


M.  M.  Siddiqui 


I.  INTRODUCTION  AND  SUMMARY 


In  recent  years  the  approach  to  time  series  has  undergone  some 
important  changes,  and  the  methods  of  analysis  have  been  modified 
accordingly. 

The  classical  approach  to  time  series  may  be  summarized  as 
follows  . We  have  observations,  x^, . . , on  a time  variable  and 

we  represent 


where  m^  is  the  deterministic  and  the  random  component  of 
at  time  t.  t=  1, , , , ,n,  are  independent  and  identically  distributed 


x^  = m^  + t = 1,  2 


, • • • , n 


2 

random  variables,  each  with  mean  zero  and  variance  cr  . The  distri- 

2 

bution  of  is  usually  assumed  to  be  normal  and  the  variance  cr  to 


be  unknown,  m^  is  represented  either  as  a polynomial  in  t 


p(t)  = Cq  + cit  + 


where  the  degree  of  the  polynomial,  k,  and  the  coefficients 


^o  » e • • 


, cj^,  are  assumed  to  be  unknown  constants;  or  m^  is 


represented  as  the  sum  of  trigonometric  terms 


1 


s(t)  = cos  X^t  + sin  + • . • + «ip  cos  V + S sin  V’ 

where  p,  a^s,  b*s  and  X*s  are  assumed  to  be  unknown  constants; 
or,  more  generally,  m^  is  assumed  to  be  the  sum  of  p(t)  and  s(t). 

The  method  of  least- squares  is  then  employed  to  estimate  the  co- 
efficients in  any  of  these  representations  and  the  periodogram,  i,  e, 

(a?  + b^)  multiplied  by  a constant,  is  used  to  test  the  significance  of 
J J 

periods  2ir/ Xj  . 

For  a summary  treatment  of  these  methods  the  reader  may 
refer  to  [10,  Chapters  29>  30], 

In  more  recent  years,  the  concept  of  stationarity  of  time 
series  has  been  playing  an  important  role.  We  may  still  try  to 
decompose  a time  series  into  a deterministic  component,  m^,  and 
a random  component,  if  there  are  reasons  to  believe  that  some 

of  the  factors  underlying  our  observations  have  been  changing  deter- 
ministically during  the  course  of  our  observation.  However,  there 
are  reasons  to  believe  that  in  many  physical  experiments  these  factors 
are  in  statistical  equilibrium,  i.  e,  changing  randomly  with  a fixed 
distribution  function,  during  the  course  of  our  observation,  or,  even 
if  they  change  more  violently,  this  change  is  random  rather  than 
deterministic.  In  any  case  the  deterministic  component,  m^,  is 
dropped,  and  a completely  stochastic  model  takes  its  place.  Several 
stochastic  models  have  been  developed  to  deal  with  non- stationary 


2 


processes  which  represent  certain  phenomena,  e.g.  population 
growth,  Brownian  motion,  epidemics,  traffic  on  telephones  and  high- 
ways, servicing  etc,  , under  specified  conditions.  We  will  not 
describe  them  here. 

In  this  report  we  will  confine  attention  to  the  techniques 
developed  in  recent  years  to  deal  with  stationary  processes,  i.  e, 
time  series  under  statistical  equilibrium.  These  techniques  are 
specially  suited  to  analyse  radio  propagation  data,  taken  over  a short 
interval  of  time  under  more  or  less  constant  conditions.  The  material 
presented  herein,  excepting  one  or  two  original  results,  is  explicitly 
or  implicitly  available  elsewhere.  However,  it  seemed  desirable  to 
present  it  here  in  a manner  suited  to  the  purposes  of  those  who  use 
Fourier  analysis  or  its  variations  in  dealing  with  time  series  data. 

It  is  interesting  to  note  that  the  classical  representation,  s(t),  of  a 
time  series,  still  occupies  a prominent  place  in  the  analysis,  although 
the  coefficients,  a‘s  and  b*s,  are  not  considered  constants  any  more 
but  random  variables. 

In  section  2 necessary  probability  background  will  be  provided. 

In  sections  3 and  4 the  spectral  analysis  of  discrete-time  and 
continuous -time  processes,  respectively,  will  be  dealt  with.  In 
section  5 the  case  of  a discrete  sample  from  a continuous -time 
process  will  be  considered.  Linear  filters,  the  estimation  of  spectral 
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densities,  and  some  examples  of  stationary  processes  will  be  pre- 
sented in  sections  6,  7 and  8,  respectively.  The  references  at  the 
end  of  the  report  include  only  such  publications  which  are  directly 
connected  with  the  subject  matter  of  this  report.  For  a more  complete 
bibliography  the  reader  may  consult  the  Bibliography  at  the  end  of  [9]. 

2.  SOME  CONCEPTS  IN  PROBABILITY  THEORY 

The  basic  idea  in  statistical  analysis  is  that  of  a sample  space. 

A sample  space,  may  be  defined  as  the  set  of  all  possible  distinct 
observations,  w,  on  a physical  experiment,  actual  or  hypothetical, 
repeated  under  essentially  similar  conditions.  Any  subset,  S,  of 
is  called  the  event  S.  An  additively  closed  family,  B,  of  subsets  of 
n is  constructed  and  a probability  measure,  P,  is  defined  on  B. 

These  concepts  are  introduced  as  a theoretical  foundation  of  proba- 
bility theory.  From  a practical  point  of  view,  the  usual  frequency 
concept  of  the  probability  of  an  event  will  be  sufficient  for  proceeding 
further. 

It  may  be  that  the  observation  on  an  experiment  is  a real  number 
or  a set  of  real  numbers,  e.g.  observing  the  pressure  and  temperature 
of  a given  volume  of  a gas,  in  which  case  Q may  be  taken  an  Euclidean 
space  of  proper  dimensions,  each  characteristic  being  a coordinate. 

In  certain  other  experiments  the  observation  may  not  be  expressed  as 
a set  of  real  numbers,  e.g.  observing  the  sex  and  color  of  hair  of  an 
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individual,  in  which  case  we  may  associate  real  numbers  to  these 
characteristics  under  some  conventional  system,  e.g.  0 for  female 
and  1 for  male,  A random  variable,  X(w),  is,  then,  defined  as  a 
real-valued  function  of  w,  i.  e,  , a function  which  associates  a real 
number  to  every  element  g)  of  A complex-valued  function  of  o) 
will  be  called  a complex -valued  random  variable.  The  subsets  of  Q 
are  associated  through  X(o))  with  the  subsets  of  the  real  line,  Rj^, 

and  hence  a probability  measure  is  induced  on  Rj.  If  S is  a subset 

\ 

of  Rj,  we  may  speak  of  the  probability  that  X(g))  belongs  to  S and 
write  Pr(X(cj)€S),  In  particular 

P(y)  = Pr(X(o))  <y). 


where  y is  a real  number,  is  called  the  distribution  function  of  X(co), 
P(y)  is  a non- deer  easing  function  of  its  argument  with  P(-oo)  = 0 and 
P(oo)  = 1.  It  can  be  a step  function  or  an  absolutely  continuous  function 
or  a mixture  of  both.  In  case  P(y)  is  absolutely  continuous  its  deriva- 
tive P'(y)  = p(y)  is  called  the  probability  density  function  or  the 
frequency  function  of  the  random  variable  X(o)).  Obviously 

y 

p(y)  — 0,  Pr(X(w)  :<  y)  = P(y)  = ^ p(x)dx, 

-oo  -CO 

The  idea  of  a vector  random  variable  presents  no  conceptual 
difficulty.  In  case  of  more  than  one  random  variable  we  speak  of  the 
joint  distribution  and  the  joint  probability  density  functions. 


I 


p(y)dy  = 1 , 
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A random  process,  {x(t,u>)},  besides  being  a function  of  u)  is 
also  a function  of  t.  We  will  speak  of  t as  a time  variable  though 
it  may  be  an  arbitrary  variable.  For  every  value  of  t,  x(t,w)  is  a 
random  variable.  Thus  a random  process  is  a collection  of  random 
variables.  For  every  fixed  o),  x(t,a>)  is  a sample  function  or 
realization  of  the  process.  Thus 

x(t,o))  = A(a>)  cos  ^\(a))t  + <j)(w) 

where  (A,  4>)  is  a vector  random  variable,  is  a random  process, 

A random  process  may  be  visualized  as  a large  number  of 
simultaneous  independent  measurements  of  the  same  experiment 
over  a long  period  of  time.  Theoretically  the  number  of  measure- 
ments is  to  be  indefinitely  large  and  the  time  of  observation  infinite 
in  both  directions.  In  practice  we  shall  be  dealing  with  a single 
realization  of  the  process  over  a finite  interval  of  time. 

The  time  variable  t may  vary  continuously  or  may  take  only 
discrete  values  for  a given  random  process.  In  the  former  case  we 
shall  call  the  process  a continuous -time  process  and  in  the  latter 
case,  a discrete-time  process.  A continuous -time  process  may  be 
made  discrete  by  considering  it  at  discrete  time  points. 

There  is  no  conceptual  difficulty  in  talking  about  a vector 
random  process  nor  in  considering  t as  a vector  variable.  In  the 
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following  discussions,  however,  we  will  confine  ourselves  to  a one- 
dimensional random  process  and  t will  always  represent  time. 
Furthermore,  the  letter  w will  not  occur  explicitly  and  we  will  write 
{x(t)}  for  a continuous -time  process  with  x(t)  as  a sample  function 
and  {x^}  for  a discrete-time  process  with  x^  as  a sample  function. 

If  u is  a random  variable,  Eu  will  be  used  to  denote  its  mean  value 
over  the  sample  space  Q, 

A real  random  process,  {x(t)}  , will  be  called  weakly 
stationary  or  stationary  if,  for  all  t and  s, 

(i)  Ex^(t)  < oo 

(ii)  Ex(t)  = Ex(0) 

(iii)  Ex(t)x(t+s)  = Ex(0)x(s). 

It  will  be  called  strictly  stationary  if  the  joint  distribution  of 
x(tj+h), . , , ,x(tj^+h)  is  identical  with  the  joint  distribution  of 
x(tj^), , . , ,x(tj^)  for  every  n,  tj,  . . . , t^^  and  every  h, 

A process  is  called  Gaussian  if  x(ti), . • . ,x(t  ) have  a joint 
normal  distribution  for  every  n,tp  . , . , t^^.  A Gaussian  process  is 
completely  specified  by  Ex(t)  and  Ex(t)x(s)  given  for  all  t and  s. 
Hence  a weakly  stationary  Gaussian  process  is  strictly  stationary. 
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3.  FOURIER  ANALYSIS  OF  A DISCRETE- TIME  PROCESS 
Let  {x^}  be  a stationary  discrete- time  process  with  Ex^=m 
and  E(xj.-m)(x^^g-m)  = Cg.  Cg  will  be  called  the  auto  covariance 
function  and  pg  = C^/Cq,  the  autocorrelation  function  of  the  process. 
Obviously,  C^g  = Cg,  p«g  = pg.  If  sCg  0 as  s oo,  then,  the 
series 

oo  o x oo 

^ « TTl  S \ ^ 

S CgO  = Cq+2  Z CgCOsZTTsX 

s=-oo  s=l 

is  a Fourier  series  of  its  sum  f(X),  so  that 


(1) 

(2) 


f(X)=  S + 2 


s=-oo 

= d\= 

“Vz  -Vz 


oo 

Z CgCOSZTTSX 
S=1 

cos  ZttsX  dX. 


We  readily  see  from  (1)  that  f(-X)  = f(X)  and  from  (2)  that 

^Vz 

\ f(X)  dX  = C^.  f(X)  is  called  the  power  spectral  density  fimction 
-*Vz 


of  the  process  {x^}  , 

Consider  the  periodogram 


n 


(3)  In(p.)  = 


n 


t=l 


.ij 

^ k=l  t=l 


Z (x^-m)(xj^-m)  , 
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We  have  Ij^(-  |jl)  = arid 


n 


n 

S 


(4)EI„(^)=^  S 

k=l  t=l 


k-t 


27ri(t"k)|jL  _ 


n-1 

S il- 

8=-(n-l) 


n 


-iZTTSjJL 


OD 

If  ZJ  sCg  < CD,  we  have 
1 

oo 

(5)  lim  E]^(n)  = S Cg  = f(ji) . 

n-*“CO  -CD 


Since  Ij^(|x)  ^ 0 , f(|i,)  > 0. 

Thus  we  may  summarize  the  properties  of  f(X.)  in  the 
following  manner: 


(i)  f(-\)  = f(X) 


(ii)  f(M  ^ 0 


(iii) 


dk  = 


i.  e,  f(k)  is  symmetric, 
i.  e.  f(k)  is  non-negative. 


so  that  f(X)/CQ  is  a symmetrical  frequency  function  over  (~Yz»  Vz) 
We  will  set  f(X)  = 0 outside  the  interval  (-Va*  Vz)  • 

W riting 


(6) 


X 

f(jJL)  dfx. 


- Vz  — — Vz  9 
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we  will  call  F(\)  the  (power)  spectral  distribution  function  of  {x^}  , 
f*(X)  = f(X)/CQ  and  F*(\)  = £{\)/C^  will  be  called  the  normalized 
spectral  density  and  the  normalized  spectral  distribution  functions, 
respectively.  Since  f(X)  is  the  Fourier  series  transformation  of 
Cg,  f*(\)  is  the  Fourier  series  transformation  of  p^.  Thus  we  have 
proved  the  following  theorem. 

Theorem  1,  If  the  Fourier  series  transformation,  f*(X),  of 
the  autocorrelation  function,  Pg,  of  a discrete-time  real  stationary 
process  exists,  then  it  is  a symmetrical  frequency  function  over  the 
inte  rval  (-]4,  ) • 

It  can  also  be  shown  that  given  a symmetrical  frequency 
function,  f*(X),  over  (-V^,  ) , we  can  find  a real  discr ete -time 

stationary  process,  such  that  the  Fourier  coefficients  of  f*(X) 

are  the  autocorrelations  of  the  process  {x^}  . 

Returning  to  the  equation  (5)  we  observe  that  if  we  have  a sample 
. ,Xj^  then  the  periodogram  Ij^(X)  is  an  asymptotically  unbiased 
estimator  of  f(X).  However,  if  we  insert  the  value  of  from  (2) 
into  (4),  we  obtain 


n-1 
s=-(n-l) 


‘/a 


2Tris(X-|j.) 


f(X)  dX 
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Now,  using  the  identities 


P p+1 

s _ z-z^ 

S z = 


s=l 


1-z 


V = z-(p+l)  + pgP'*'^ 


Z)  sz  = 
s=l 


with  p = n-l,  and  z=e^^^^^^  we  can  sum  the  series  in  the 
brackets  and,  using  certain  trigonometrical  identities,  we  finally 


obtain 


^2 


.n(|j.-H4)  . 2 

'r*  y sin  7TV 


(71  t>l  ■ f HO) -- r 

" J n sin"^Tr(X-|jL)  n^  J “ sin^(7rv/n) 

-/a 


Hence  In<P)  i®  an  unbiased  estimate  of  a 'filtered'  spectral  density 
fn(p)  for  finite  n.  Only  when  n— *-oo,  fj^(jjL) f(p). 

For  convenience  we  suppose  that  n=  2N+1  and  represent  the 
sample  Xp  . . . ,Xj^  by  the  trigonometric  series 

(8)  = a^  + S ^aj  cos  + bj  sin  ^ » t - 1,  2,  . . . , n. 


The  coefficients  a's  and  b's  are  determined  by  the  usual  method  of 
harmonic  analysis,  i.  e.  , by  the  relations 


(9) 


a_  = — S x^  = X 


n 


27rjt 

f 

j~l,8*»  , N , 

n 

27rjt 

» 

n 

j“*l,»»«,N, 
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where  the  summations  axe  with  respect  to  t from  1 to  n and  where 
c is  an  arbitrary  constant.  Thus,  setting  c = m,  we  have 


(.0) 


2 2 

- i (^S(x.  - m)  cos  + — (^S(x.  - m)  sin 

n\t  n/n\  t n/ 


i Zxrjt  2 

= — I S(x. -m)  e ^ 
n ^ 


= In  (-)  . j = 1. . . . * N 
“ n 


1 , ,2  2 

However,  Ij^(O)  = ” | S(x^-m)  | = n(aQ-m)  , and  depends  on  the 

knowledge  of  the  process  mean  m.  If  we  replace  m by  x every- 
where, Ij^  (^)  will  not  change  for  j=l,  ...,N,  but  1^(0)  s 0 and 

\ 

no  estimate  at  zero  frequency  will  be  available.  It  seems  desirable, 
therefore,  to  estimate  the  process  mean,  m,  by  a much  longer 
sample  preferably  from  the  observations  which  do  not  include 

Xji^ , . . . , 

Thus  the  representation  (8)  provides  the  periodogram  estimate 
of  the  spectral  density  function  at  the  frequencies  j/n,  j = 1, . , , , N, 
in  terms  of  the  coefficients  aj  and  bj  . 

We  have 

(11)  Ea^  =m, 

2 2 

var  = E(ao-m)  = ^ E | S(xj-m)  | = i £1^(0) 

n ^ 


12 


Also,  for  j / 0, 


E a. 
J 


var  a. 
J 


Eb,  = 0 
J 

2 4 


n n 


Ea!"  = E S S (x  -m)(x  -m)  cos  cos 

n t=l  s=l  n n 


n n 


— S sc.  [cos 
7 t-  S 

t=l  s=l 


2TTj(t+s) 


+ cos 
n n 


2 l^l  2Trjr  , 2 ^ ^ 47rjs 

— E I - - — - cos  — ^ S cos — s?— 


n _ / i.\  n 

r=-(n-l)^ 


n 


2 o 
n s=l 


n 


n-1  n-r 

+ 4 S S c,  cos  islis+li 
r=l  s=l 


The  first  summation  is  seen  to  be  eqiial  to  2/n  EIj^(j/n)  = 2/n  fjj(j/n). 
Using  the  trigonometrical  identity 
P 

S cos  (a+2sp)  = cosec  p sin  (pp)  cos  {a+(p+l)p} 
s=l 


with  p=  27rj/n,  a=  2‘irjr/n,  we  find 


n-r 


s cos  2li(2s+r)  _ ^ 3i„  ML 

s=l 


n 


n 


n 


Hence  for  j f 0, 


(12) 


,=£f  (l)-Acot2’U 


n-1  . 

S C siniai. 
n , r n 

r=l 


= 2f  (i)  + o(4).  since  S C_  — 
n n n ^ ^ 


oo 


n‘ 


r=l 


sin  — - — < CD 
n 
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It  can  be  shown  similarly  that 


(12a) 


var  b.  = — f (^)  + cot 


'j  ' n n V ^ 


n-1  - . 

_ Zirir 

S C sin  — — 
n , r n 

r=l 


= ^f  (h  + o(i). 

n n n n^ 


Ea^aj  = 0(Jj).  Eb^bj  = OC-lj),  j. 


Ea.b.  = 0(-!-),  alii,  j. 
^ J n^ 


Thus  the  correlation  between  any  two  coefficients  is  of  order  l/n  • 

Now,  if  {xj.}  is  a Gaussian  process,  aj  and  bj  will  be 
normally  distributed  and  since  different  coefficients  are  approximately 
uncorrelated,  they  may  be  assumed  to  be  independent  of  each  other. 
Thus  the  Ij^(j/n)  are  also  approximately  independent  of  each  other  and 

2InO/n)  / fnOA) 

is  approximately  a variate  with  2 degrees  of  freedom.  We, 
therefore,  have 

(13)  var  Ij^(j/n)  ^ fj^^(j/n),  j = 1, , , , , N . 

Even  if  the  process  is  not  Gaussian,  the  result  (13)  can  be 
proved  to  be  valid,  (13)  shows  that  the  periodogram,  has 

standard  error  equal  to  (in  fact  a little  greater  than)  EI^(X),  which 
makes  it  an  undesirable  estimate. 
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We  will  discuss  some  proposed  modifications  of  in 

section  7, 


4.  FOURIER  ANALYSIS  OF  A CONTINUOUS- TIME  PROCESS 
Let  {x(t)}  be  a continuous -time  real  stationary  process  with 
Ex(t)  = m and  E [x(t)-m]  [x(t+s)-m]  = C(s),  p(s)  = C(s)/C(0),  If  p(s) 
is  continuous  at  s = 0 then  it  can  be  shown  that  it  is  continuous  for  all 
s and  also  that 

lim  E I x(t+s)  - x(t)  1^=0, 
s-*o 

Consider  the  periodogram 


(14)  = j J [x(t)-m]e 


ZTritX. 


dt 


T T 

^ [ x(t)-m][  x(k)-m]e^^^^^  ^^\ikdt. 
o o 


We  have 

(15)  gT(M=EIx(\)=  j*  = ^ (^"^)  C(8)e■^^’'®^is. 

o o -T 


Since  I,p{X)>  0,  it  follows  that 

(16)  J J p(k-t)e^^’'^*”^^^dkdt  > 0. 

o o 


Since  jp(s)|<  1,  p(0)  = l,  p(s)  is  continuous  and  symmetric  and  the 
integral  in  (16)  is  non-negative,  therefore  [5,p.  91]  we  have  the 
following  theorem. 
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Theorem  2,  If  the  autocorrelation  function  p(s)  of  a continuous- 


time real  stationary  process  is  continuous  at  s = 0,  it  is  the  char- 
acteristic function  of  a symmetric  distribution,  i.  e. 


where  G*(X.)  is  a distribution  function  of  a symmetric  distribution 
over  (-00,  00) . 

Conversely,  it  can  be  shown  that  for  a symmetric  distribution 
dG*(X)  there  exists  a real  continuous -time  stationary  process  such 
that  the  characteristic  function  of  dG*(X)  is  identical  with  the  auto- 
correlation function  of  the  process. 

If  G*(X)  is  absolutely  continuous  and  has  a frequency  function 
g*(X)  = dG*/dX,  then  it  follows  that 


and  that  g*(X)  is  continuous.  Writing  g(X)  = C(0)g*(\),  we  have  the 
transform  pair 


(17) 


(18) 


CO 


(19) 


00 


(20) 


00 
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Since  C(-s)  = C(s),  we  have  g(-X)  = g(X);  also  g(X)  > 0 as 


g*(X)  is  a frequency  function,  and 

\ g(X)dX=C(0). 

“OO 

g(X)  is  called  the  power  spectral  density  function  of  the  process  {x(t)}, 

pOO 

If  \ sC(s)ds  exists,  then  from  (15)  it  follows  that 


lim  g-T.(X)=g(X) 
T— oo 


However,  if  we  insert  the  value  of  C(s)  from  (19)  into  (15),  we 


obtain 

(21) 


gx(ix)  = - M)g(X)e'2’^<^-^^>^dXds 

-T  -GO 


= j*  g(X)dX  j"  (l  - ei2TT(X-M.)s 


-OO 


-OO 


”g(X)  ^in^^T(X-jx) 


ir2T(X-|jt)2 


=rr<-o 


gifi ) + g(  ^ 


. 2 

Sin  TTV 

TT^V^ 


dv. 


Thus,  if  we  have  a continuous  sample  over  (0,T),  then  the  periodogram, 
unbiased  estimate  of  g'p(p)  which  is  a 'filtered'  spectral 

density. 
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Now,  the  sample  function  over  (0,T)  can  be  represented  by  a 


Fourier  series,  i.  e.  , 


(22) 


oo  / . . \ 

x(t)  = + S I aj  cos  + bj  sin  J , 0 < t < T . 

j=lV  / 


We  then  have 


(23)  a^  = 


dt  = X , 


a = 

j 


” ^ J x(t)  cos  dt  = |;  J [x(t)-c]  cos  dt,  j 0, 


b.  = 
J 


\ ^ x(t)  sin  dt  = |;  y [x(t)-c]  sin  dt, 


where  c is  an  arbitrary  constant.  Setting  c = m,  we  have 

T i2.Tr}t  2 


Thus  the  representation  (22)  provides  the  periodogram  estimate 
of  g»p(M  at  the  frequencies  X = j/T,  j = l,2,.,.,  in  terms  of  the 
Fourier  coefficients  aj  and  bj  and  from  (23)  we  see  that  these  esti- 
mates are  independent  of  the  process  mean  m.  However, 


T 

(0)  = ^ j J [x(t)-m]dt 


= T(aQ-m)  , 
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and  depends  on  the  knowledge  of  m.  If  we  replace  m by  x then 


I-p(O)  = 0;  hence,  for  an  estimate  at  X=  0,  we  require  the  knowledge 
of  m independent  of  the  sample  mean. 

With  calculations  similar  to  those  carried  out  for  the  discrete- 
time case  [see  also  12  pp.  157-160]  we  can  verify  that 

(25)  Ea^  = m,  Ea^  = Ebj  = 0,  j ^ 0, 

var  a^j=  i (0), 

var  a.  = ^ (i)  + 0(-L) 

Ea.a.  = O(i-).  Eb.b,  = O(-i-),  i / j, 

■‘■J  J-J  rj,  ^ 

Ea>b^  “ O ( y ) » all  i»j. 

^ j •j’ 

Also 

(26)  var  I^(\)^  g^(X),  X = j/X,  j / 0. 

5.  FOURIER  AJMALYSIS  OF  A CONTINUOUS- TIME  PROCESS 
OBSERVED  AT  DISCRETE  TIMES 

The  most  interesting  case  is  one  in  which  we  have  a continuous 

record  over  (0,  T)  as  in  section  4 but  select  a systematic  sample 

x(a),  x(a+6),  x(a+26),  x(a+n-15),  where  0<  a<a  + n-16^T, 
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We  may  then  consider  two  processes  {x(t)}  and  {x^}  where 
x^  = x(a  + 1-1  6),  t = 0,  +1,  4^2,  ....  We  therefore  have  C^=C(t6). 
If  f{\)  and  g(X.)  are  the  spectral  densities  of  {x^}  and  {x(t)} 
respectively,  we  have 


}/z 


pOO 

= j g(X) 


iZTTtSX 
e dX 


-CD 


oo 


r=  -oo 


J g(V/6) 


-v^ 


Vz 


iZirtX 


OO 


6 f 8 

r=-oo 


(¥) 


d\ 


for  every  integer  t.  From  the  uniqueness  of  Fourier  coefficients 
it  follows  that 


(27) 


The  frequencies 

X + 1 X + 2 


are  called  aliases  to  X/6  as  they  become  indistinguishable  from  x/S 
when  we  consider  the  discrete -time  process  {x^}  instead  of  the  con- 
tinuous time  process  {x(t)}. 
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If  we  set  a = 6 and  n6  = T , we  have 


oo 


(28) 


f (i)  = i s g 


n n 


r=-oo 


j+nr 

T 


oo 


^ g(^)  s 

r =1 


gi^)+g(^ 


n . n 
— 1 < — 
9 — J — 9 


c°° 

Since  V g(x+jji)  djx  exists  for  every  x,  we  have 


o 


^ g(x+|j,)  d|jL  -*•  0 as  n oo  . 

n/T 

oo  / +k  \ 

Now  1/t  Z)  g f x+ j is  an  approximating  Riemann  sum  to 

k=o  ^ ^ 

the  integral  with  A|jl=1/T  and  hence  tends  to  zero  as  n-*-  oo  . 
Therefore,  if  n is  large,  we  have  for  small  values  of  j 


(28a) 


-f(^)  = 

n n T T 


In  practice,  when  we  are  analysing  a record  with  superposed 
undesired  ’noise'  such  as  due  to  recording  or  reading  instrument  we 

^n-r 


will  select  6 or  n in  such  a way  that  g 


2T 


is  dominated  by  the 


noise  spectral  density  at  the  frequency  (n-l)/2T.  Here,  we  are 
assuming  that  the  noise  spectral  density  is  negligible  as  compared  to 
the  spectral  density  of  our  data  in  low  frequency  ranges.  We  are 
then  not  interested  in  the  value  of  g(X)  for  X.>(n-1)/2T,  or,  perhaps. 
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the  values  of  g(X)  for  X>(n-1)/2T  are  considered  negligible.  In  this 
situation  the  effect  of  aliasing  will  be  negligible  except  at  the  point 
X = (n-1)/2T,  where  it  will  make  our  estimate  twice  as  large  as  it 
actually  should  be. 


6.  LINEAR  FILTERS 


If  we  write  a*(X)  for  the  Fourier  transform  of  a function 


a(t)  and  if 


pOO 

a(t)  = \ b(t-u)  c(u)  du  , 

-CD 


then  it  is  easy  to  show  that 


a*(X)  = b*(X)c*(X). 


a(t)  is  called  the  convolution  of  b(t)  and  c(t) . 

Let  {x(t)}  and  {z(t)}  be  two  continuous -time  real  stationary 
processes  such  that 


(29) 


r°° 

x(t)  = \ W(t-u)z(u)du 
-oo 


pOO 

V W(u)  z(t-u)  du  , 


where  W(t)  is  a real  function  integrable  over  (-00,  00).  W(t)  is 
called  the  linear  filter ^ z(t)  the  input  and  x(t)  the  output  of  the 
filter  W(t)o  if  denote  the  auto  covariance  and  g^{\) 

and  g^(\)  the  spectral  densities  of  the  processes  {x(t)}  and  {z(t)} 
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respectively,  and  W*(X;)  is  the  Fourier  transform  of  W(t),  then  we 
have 


.00  ^00 


C^(s)  =Ex(t)x(t+s)  =E 


n 


-00  -00 


W (u)W(v)z(t-u)z(t+s-v)dudv 


00  _oo 


I 


-00  -00 


W(u)W(v)  C (s+u-v)dudv 
z 


g^(M 


( 


-00 


00 


2 TTl  S X / \ j 

e C^(s)ds  = 


00  ^.00  ^00 


-00-00  -00 


i-ZTrisX 


W(u)W(v)C  (s-u-v)dudvds. 
z 


Making  the  transformation  s'  = s+u-v,  u=u,  v = v,  we  have 

Joo  00  00 

^ C^(s')ds'  ( e^’^^W(u)dur  W(v)dv 

-00  -00  -00 


= gJM  |W*(X)|^.  -00  < X < 00  . 

z 


Now,  if  the  filter  is  known  and  the  spectral  density  of  one  of  the 
processes  is  known,  then  the  relation  (30)  gives  the  spectral  density 
of  the  other  process.  It  is  sometimes  possible  to  adjust  the  filter 
in  such  a way  that  the  output  x(t)  is  approximately  a white  noise, 
i.  e.  , has  a constant  spectral  density  over  a wide  frequency  range. 
The  use  of  such  a filter  on  z(t)  is  called  the  pre whitening  of  z(t) 
and  is  a powerful  method  in  obtaining  a reliable  estimate  of  its 
spectral  density. 
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If  in  (29)  we  replace  x(t)  by  etc*  and  integrals  by  summations, 
we  obtain 

-i/a  ^ X < . 

where,  now, 

CD 

W*(X)  = S Wj^  g-i2irkX  ^ 

k=-oo 

For  further  study  on  this  subject  see  [12,  Ch,  5], 

7.  ESTIMATION  OF  SPECTRAL  DENSITY 
The  purpose  of  Fourier  analysis  of  a stationary  process  is  to 
construct  an  estimator  of  the  true  spectral  density  function  which  has 
at  least  two  desirable  properties  of  being  unbiased  and  consistent. 

It  turns  out  that  no  estimator  of  g(\)  or  f(X)  exists  which  is  unbiased 
for  finite  sample  size,  although  asymptotically  unbiased  estimators 
are  available.  The  periodogram  is  one  such  estimator.  Unfortunately, 
the  variance  of  the  periodogram  remains  bounded  from  below  no  matter 
how  large  a sample  we  take,  i.  e.  , it  is  an  inconsistent  estimator  of 
g(X.)  or  f(X).  We  therefore  try  to  construct  some  other  estimators 
which,  besides  being  asymptotically  unbiased,  are  consistent.  In 
fact,  several  such  estimators  have  been  proposed  and  a general 
method  of  'filtering*  the  periodogram  is  available  which  produces  such 
estimators.  We  will  only  mention  two  such  modified  periodogram 
estimators,  one  suggested  by  Bartlett  and  the  other  by  Tukey. 
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As  Burning  Ex^  = 0 • and  writing 


n-k 

S 

(32)  , k=  0,1, . . . ,n-l,  c_j^=  Cj^  , 

n-k 

we  observe  that 

(33)  L(M=  S fl- J^VkCOsZTi-kX,  X=i,  j = 

k=-(n-l)^  ^ 2 


Here,  ln(0)  can  only  be  obtained  when  the  process  mean  is  known  to 
be  zero.  Otherwise,  Ijj(O)  will  remain  arbitrary  depending  on  our 
estimate  of  the  process  mean. 

Bartlett  suggests  [1,  also  9»  p.  146] 


m 


(34)  I„^^Nx)=  S (l- 


k=-m 


m 


ci^  cos  ZirkX,  X.=  — , j = 0,l, 
^ m 


m-1. 


where  m=0(n)  but  m — ^ oo  as  n—^co.  Here,  the  sample  covariances 

of  large  order  have  been  avoided  as  they  are  subject  to  great  sampling 

fluctuations.  It  is  easy  to  see  that  lim  EI^^^^(X)  = f(\). 

n— ►oo 

The  variance  is  given  by 


(35) 


var 


(X) 


rsj 


r ^f^(X).  X;^  0 

|^f2(0).  X=0. 
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Tukey  has  proposed  [ 9,  p.  149;  also  2] 


m-1  ✓ V 

(36)  ^ (k)  = Cq  + 2 S 46  cos  ^ + . 54^  Cj^  cos  irkX 

k”l 


+ (,  46  cos  TT  + . 54)  cos  TrmX., 


X = -i-,  j = 0,1, . . . ,m-l, 
m 


as  an  estimate  of  f(X),  where  m=0(n)  but  m-^oo  as  n-^oo. 

is  also  an  asymptotically  unbiased  estimator  of  f(X),  and 
its  variance  is 


(37) 


var 


(X)^ 


^f^(V)[(.46)^  + 2(.54)^],  \J^0 
I ^ f^(0)[  (.46)^  + 2(.  54)^],  \=  0. 


Tukey  has  also  suggested  a method  of  setting  confidence  bands  on  f(X), 

(2)  7 

by  showing  that  '(X)  is  approximately  a distribution  with 

approximately  2n/m  degrees  of  freedom,  [2],  However,  1^^'  '(Xj) 

17.\ 

and  In  '(X2)  are  not  independent  if  | X2-X2I  — 3/m,  so  that  the  total 
degrees  of  freedom  do  not  exceed  the  available  degrees  of  freedom  n. 

If  the  sample  mean  is  subtracted  from  data  before  any  further 
analysis,  then  as  we  have  seen  In(®)  = difficult  to  give  any 

meaning  to  Ij^^^^O)  and  Ij^^^^(O)  in  such  a situation. 
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8.  EXAMPLES 


(i)  White  noise.  Let  be  a stationary  process  such  that 

E^t^t+s  — ^s  “ ® s ^ 0,  Then,  from  (1),  we  have 

f(\)  = = (Tg^, 

A process  which  has  a constant  spectral  density  is  called  a 
white  noise«  Conversely,  if  f(X)  = C,  -V2—  V2>  we  have  from  (2), 

= C,  Cg  = 0,  s / o. 

If,  besides  being  uncorrelated,  and  are  independent 

for  all  s^O,  the  discrete  time  process  called  a pure  white 

noise  [9,  p.  42], 

From  equation  (19)  we  observe  that  we  cannot  set  g(\)  = 
constant  in  a continuous  time  process  as  this  would  imply  that  C(0) 
was  unbounded.  However,  by  allowing  the  existence  of  Dirac's 
delta  function  and  its  Fourier  transform,  some  authors  have  extended 
the  concept  of  white  noise  to  continuous  time  processes.  We  may, 
however,  set  g(X.)  = constant  over  a wide  frequency  band  in  some 
cases  without  making  C(0)  infinite. 

(ii)  Moving  average  and  autoregressive  processes.  Consider 

P 

(38)  + . . . + ipy^.p  = S a,^yt-k  ' 

k=o 
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where  and  are  stationary  processes  and  a^^  are  real  constants. 
Since  is  the  convolution  of  and  y^,  where 

= a^  if  0 < t ■<  p 
= 0 otherwise, 


we  obtain 
(39) 


f (M  = 

X 


S aj^e 
k=o 


-iZirkX 


tyW 


If  y^  is  a white  noise,  x^  is  called  a moving  average  process.  On 

P 

the  other  hand  if  is  a white  noise  and  S = 0 has  all  the 

k=o 


roots  within  the  unit  circle  |z|  =1  in  complex  plane,  y^  is  called  an 
autoregressive  process  of  order  p.  In  this  case  x^  is  independent 

of  Yt-1*  Vt-2’  •••  [9.  p.  38]. 

Special  cases,  (a)  Simple  moving  average.  If  we  set 

aj^  = l/(p+l),  k=  0,  1,  , . . , p,  in  (38)  we  obtain 


(40) 


^t  *■ 


1 p 

J=o 


If  y^  is  a white  noise,  the  above  scheme  is  called  a simple  moving 
average  scheme.  Since 


p 

1 - e"^^P'''^*“ 

g*i(p+l)/2  sin(p+l)o)/2 

sin  (p+1)  0)/ 2 

^ & 
i=o 

^-io)/2  gj_j^  ^^2 

sin  u)/2 

we  have  from  (39) 


1 


. 2 


^ (p+1)^  sin'^  irX  ' 


However,  since  is  a white  noise,  fy(M  = cr^  and  from  (40) 


and  hence 


(41) 


fy(X)  = 0-y^  = (p+l)  <r^,  , 


y M (P+^ cr^2  . X < 1/3  . 


(p+1)  sin^  ttX. 


2„a  X 


In  estimating  f (\),  then,  we  only  estimate  (r  by  the  usual  method 
of  estimating  a variance, 

(b)  Markov  scheme.  We  consider  the  first-order  autoregressive 
scheme  (Markov  scheme)  in  the  form 


(42) 


yt  = pyt-i  + ^ , 


where  | p | < 1,  and  is  a white  noise.  From  (39) 


fy(X)  = fx(X.)/|l- (1- 2p  cos  27t\+ p^)  , k ^ Va  , 


since  • Since  x^  is  independent  of  y^.]^ » we  obtain  from  (42) 


(T  = 


2 2 2 

p (T  + (T 

^ y 


X 


or 


(T 

X 


Oy  (1  - P ) 
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Thus 


(43)  fy(\)  = <ry^(l-p^)a  -2pcos2irX+  p^)"\  \ ^ . 

oy  and  p are  estimated  by  the  sample  variance  and  first  serial 
correlation  coefficient  respectively. 

(c)  Yule  scheme.  Consider  the  second-order  autoregressive 
scheme  (Yule  scheme)  in  the  form 


(44) 


^t  = “^t-l  ■ Pn-2  + ="t  ' 


2 

where  is  a white  noise  and  the  roots  ofz  -oz  + 6=0  lie  within 
t 

the  unit  circle  |z|  =1  in  the  complex  plane.  From  (39)  we  have 


fy(M  = 


fx(M 

2 2 

1 + a + p ~ 2a(l+p)  cos  2irX  + 2p  cos  4ttX 


» “ Vz  — X-  — V2  • 


Now,  since  x^  is  independent  of  ^ and  y^  multiplying  the 
equation  (44)  by  and  y^  ^ respectively  and  taking  expectations, 

we  obtain 


^1  “ ^^0  " P^l» 

C2  ““  ctCj^  — pc 

or 

PX  = a - PPi* 

P2  = apj  - p , 

so  that 

(45) 

Pl(l  - pz) 

2 

PI  - P2 

2 ’ P 

1 2 ' 

1-  Pi 

1-  Pi 
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If  p denotes  the  multiple  correlation  coefficient  between  and 
(Yt-1*  Yt-2^»  given  by 


(46) 


(1-  p^)  = (1  -pj^) 


-1 


1 Pi  Pz 

Pi  1 Pi  • 

Pz  Pi  ^ 


With  simple  evaluation  we  then  obtain 


2 2/1  2. 

~ * P ) • 

Since  f^iM  = ^ » -Vz  ^ \ ^ Vz  , we  finally  have 

2 2. 

0-  (1-p  ) 

(47)  f (X)  = ■■  ■ y . ->/4<  X < 1/2. 

1 + a +p  - 2a(l+p)  cos  2ttX+ 2p  cos  4Tr\ 


To  estimate  p,  a,  and  p,  we  need  estimates  of  ay  , p]^  and  p2« 
These  estimates  are  provided  by  the  sample  variance  and  the  first 
and  second  serial  correlation  coefficients  respectively. 

(iii)  Processes  related  through  a differential  equation.  For 
continuous -time  processes  we  may  consider  the  differential  equation 

(48)  a + a , f + . . . + a y = x(t)  . 

P dtP  dtP-1  ° 


To  obtain  a relationship  between  the  spectral  densities  g^(X.)  sind 
gy(X.)  in  this  and  similar  cases,  we  outline  a technique  suggested  by 
Cramer  [3,4]  of  representing  the  process  as  a Fourier  transform  of 
an  orthogonal  set  function. 
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If  and  are  two  disjoint  sets  on  a real  line,  a set  function 
z(I)  is  called  orthogonal  if 

Ez(Ij^)  = 0 . 

Here  z denotes  the  complex  conjugate  of  z.  Writing 

dz(|x)  = z(djji)  , 

Cramdr  shows  that  every  stationary  continuous -time  process,  {x  (t)}, 
may  be  represented  as 

(49)  e^27TtX.^^(^j 

-oo 

where  is  orthogonal  set  function  with  E|dz^(X)|  = dG^(X), 

and  G^(X.)  is  the  spectral  distribution  function.  If  x(t)  is  real,  then, 
also 

pOO  

x(t)  = J dz^(|x)  . 

-oo 

Thus 

C^(s)  = Ex(t)x(t+s)  = E ^+12Tr(t+s)^ 

-OO 

which  is  the  same  as  (19)  in  the  case  G^(X)  is  absolutely  continuous. 
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Returning  to  (48)  let  us  represent 


r°°  izirtx. 

y(t)  = \ e dZy(\) 

-oo 

x(t)  = ( gi^TTtX.  . 

-CX) 

Then,  from  (48) 

dZy(X)  = J dz^(M  . 

-oo 


I 


.00 


iZirtX 


p 1 

S aj^(27riX) 
k=o 


From  the  uniqueness  of  the  Fourier  transform,  we  have 


S aj^(2Tri\) 
k=o 


dZy(X)  = dz^(M  . 


Multiplying  by  the  complex  conjugates  and  taking  expectations, 


we  obtain 


^ k 

S aj^(2iriX) 

k=o 


2 

dGy{X)  = dG^(X) 


or 

(50) 


gJM  = 


^ k 

S aj^(2iriX) 

k=o 


2 

gy(M  • 


A representation  similar  to  (49)  for  a discrete-time  process, 
{x^},  is 
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1/^ 

(51)  x^  = J e^"*^dz^(X), 

where,  now,  E|dz^(\)|  - dF^(\),  Edz^(\) dz^((j,)  = 0,  X / n,  and 

F (X)  is  the  spectral  distribution  function  of  {x, } . 

X t 

(iv)  Shot  noise.  Let  us  consider  instants,  tj^,  distributed  at 
random  on  the  time  axis  according  to  a Poisson  process  of  density 
p > 0.  Let  n(t2  - tj)  be  the  number  of  these  instants  belonging  to  the 
interval  (t|,  t^) . Writing  dn(t)  = n(dt),  it  is  then  known  that 

Edn(t)  = pdt,  E[dn(t)]^  = pdt,  E [dn(t)dn(s)]  =0,  t / s, 

so  that  n(*  ) is  an  orthogonal  set  function. 

Consider  the  process 

(52)  x(t)  = S a(t-t.)  , 

tj<t  ^ 

where  a(t)  is  a real  function  with  a(t)  =0  if  t < 0,  and  a(t)  is 
integrable  over  (0,  oo) . 

Equation  (52)  may  also  be  written 

pCX> 

x(t)  = \ a(t-s)dn(s),  dn(s)  = n(ds) . 

-00 
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We  have 


pOO 

pOO 

Ex(t)  = P \ a(t-s)ds 

= p \ a(s)ds=m,  say. 

-00 

0 

Also, 

^00 

C 

C(u)  = E [x(t)-m]  [x(t+u)-m] 

= V a(t-s)  a(t+u- s)  var  dn(s) 

-cx> 

pOO 

= p \ a(t' ) a (t'+u)  dt*  . 

-CD 


The  right  hand  side,  except  for  the  factor  p,  is  the  convolution  of 
a(u)  and  a(-u).  Hence,  if  a*(X.)  is  the  Fourier  transform  of  a(t), 
we  obtain 

(43)  g^(M  = p|a*(X)|^. 


.CD 


Thus  x(t)  is  composed  of  a 'direct  current'  p ^ a(t)dt  and  a 'shot 

o 

2 

noise'  with  continuous  spectral  density  p|a*(X)| 

Special  case,  a(t)  = -< 


fie-^/T  t-0 

T 


0 


t < 0 . 


|a*(X)|^  = 7— 


gA\)  = % 


X,  o P 

, Ex=  p,  var  X = 

2T 


For  further  examples  see  [12,  chs.  3,4,5], 
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